Method for rendering 2D and 3D data within a 3D virtual environment

ABSTRACT

One variation of a method includes: accessing a 2D color image recorded by a 2D color camera and a 3D point cloud recorded by a 3D depth sensor at approximately a first time, the 2D color camera and the 3D depth sensor defining intersecting fields of view and facing outwardly from an autonomous vehicle; detecting a cluster of points in the 3D point cloud representing a continuous surface approximating a plane; isolating a cluster of color pixels in the 2D color image depicting the continuous surface; projecting the cluster of color pixels onto the plane to define a set of synthetic 3D color points in the 3D point cloud, the cluster of points and the set of synthetic 3D color points representing the continuous surface; and rendering points in the 3D point cloud and the set of synthetic 3D color points on a display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a continuation of U.S. patent application Ser. No.17/107,647, entitled “METHOD FOR RENDERING 2D AND 3D DATA WITHIN A 3DVIRTUAL ENVIRONMENT,” filed on Nov. 30, 2022, which is a continuation ofU.S. patent application Ser. No. 16/379,728, entitled “METHOD FORRENDERING 2D AND 3D DATA WITHIN A 3D VIRTUAL ENVIRONMENT,” filed on Apr.9, 2019, which claims the benefit of U.S. Provisional Application Ser.No. 62/655,116, entitled “METHOD FOR RENDERING 2D AND 3D DATA WITHIN A3D VIRTUAL ENVIRONMENT,” filed on Apr. 9, 2018, each of which isincorporated by reference in its entirety herein.

TECHNICAL FIELD

This invention relates generally to the field of autonomous vehicles andmore specifically to a new and useful method for rendering 2D and 3Ddata within a 3D virtual environment in the field of autonomousvehicles.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a graphical representation of a first method;

FIG. 2 is a graphical representation of one variation of the firstmethod;

FIGS. 3A-3D are graphical representations of one variation of the firstmethod;

FIG. 4 is a flowchart representation of a second method;

FIG. 5 is a graphical representation of one variation of the secondmethod; and

FIG. 6 is a graphical representation of one variation of the secondmethod.

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.Variations, configurations, implementations, example implementations,and examples described herein are optional and are not exclusive to thevariations, configurations, implementations, example implementations,and examples they describe. The invention described herein can includeany and all permutations of these variations, configurations,implementations, example implementations, and examples.

1. First Method

As shown in FIGS. 1 and 2 , a first method S100 for rendering 2D and 3Ddata within a 3D virtual environment includes: generating the 3D virtualenvironment in Block Silo; defining a vehicle origin representing anautonomous vehicle in the 3D virtual environment in Block S120; defininga LIDAR origin in the 3D virtual environment based on a known (orestimated) position of a LIDAR sensor on the autonomous vehicle in BlockS122; defining a camera origin in the 3D virtual environment based on aknown (or estimated) position of a camera on the autonomous vehicle inBlock S124; calculating a ray passing through the camera origin andrepresenting a principle axis of the camera in Block S130; defining animage plane normal to the ray and offset from the camera origin in ahorizontal plane by a target offset distance in the 3D virtualenvironment in Block S132; projecting a sequence of 2D images recordedby the camera onto the image plane in Block S140; and rendering aconcurrent sequence of 3D LIDAR frames, recorded by the LIDAR sensor,relative to the LIDAR origin within the 3D virtual environment in BlockS142.

1.1 Applications

Generally, the first method S100 can be executed by a system (e.g., aremote computer system, a computer network, and/or an annotation portalexecuted on a local computing device) to simultaneously renderconcurrent 3D LIDAR and 2D image data—recorded by sensors on avehicle—within a single virtual environment to enable a human annotator,vehicle technician, or rider to quickly ascertain alignment betweenthese 3D LIDAR and 2D image data and to more easily comprehendcorrespondence between the 3D LIDAR and 2D image data. In particular,rather than stitching multiple concurrent 2D image images into a 3Dcolor image and aligning this 3D color image with a concurrent 3D LIDARimage—which may be processing-intensive, data-intensive, and/ortime-consuming for a local computer or computer network to execute thesystem can instead: define a 3D virtual environment; define 2D imageplanes at key positions within the 3D virtual environment; render 3DLIDAR images in sequence within the virtual environment; and projectconcurrent 2D images onto corresponding 2D image planes in the virtualenvironment.

Therefore, to enable a human annotator (or a technician, or a rider, asdescribed below) to visually interpret 3D LIDAR data from a LIDAR sensorarranged on the autonomous vehicle, the system can present these 3DLIDAR data within a 3D virtual environment. To further improve the humanannotator's comprehension of these 3D LIDAR data, the system can define2D image planes within this 3D virtual environment and project 2D colorimages onto these 2D image planes based on known (or estimated) relativepositions of LIDAR sensors and color cameras on the autonomous vehicle,thereby fusing 3D point and 2D color data within one, visuallydiscernible virtual environment.

In one implementation, the system accesses 3D LIDAR frames (e.g., 3Dpoint clouds) and 2D images (e.g., 2D image images) recorded by anautonomous vehicle (or by a manually-operated vehicle outfitted withLIDAR and color cameras) and implements Blocks of the first method S100to compile these 3D LIDAR and 2D images into one 3D environment withinan annotator portal in which a human annotator manually labels objectsrepresented in these 3D LIDAR and 2D image data. The system can definean origin in the virtual environment at a representation of theautonomous vehicle such that external objects—such as other vehicles,lane markers, road signs, and trees, etc.—are depicted in the virtualenvironment as moving relative to the representation of the autonomousvehicle. For each camera on the autonomous vehicle, the system candefine an image plane that is perpendicular to the ground plane in thevirtual environment, normal to the principle axis of the camera in ahorizontal plane, and offset from an origin of the camera—represented inthe virtual environment—by an offset distance. In this implementation,the system can set this offset distance such that an image plane definedfor a 2D image feed recorded by a laterally-facing camera (e.g., aleft-forward camera, a right-forward camera) on the autonomous vehicleis approximately centered on an adjacent road lane represented in thevirtual environment.

By then projecting 2D image feeds from these laterally-facing camerasonto their corresponding image planes while rendering concurrent 3DLIDAR data (e.g., concurrent 3D point clouds recorded by a set of LIDARsensors on the autonomous vehicle) within the virtual environmentaccording to Blocks of the first method S100, the system can presentclusters of points passing through an image plane while 2D imagesprojected onto the image plane depict a vehicle moving past the field ofview of the camera that recorded these 2D images, as shown in FIGS.3A-3D. The human annotator viewing the virtual environment may quickly,visually discern a correlation between this cluster of points and thevehicle depicted in this sequence of 2D images projected onto the imageplane given alignment between these 3D and 2D data within the 3D virtualenvironment. Accordingly, the human annotator may quickly select andlabel this cluster of points as a vehicle.

1.2 Autonomous Vehicle and Data Collection

The first method S100 can be executed by a computer system (e.g., aremote server, hereinafter the “system”) in conjunction with anautonomous vehicle. The autonomous vehicle can include: a suite ofsensors configured to collect information about the autonomous vehicle'senvironment; local memory storing a navigation map defining laneconnections and nominal vehicle paths for a road area and a localizationmap that the autonomous vehicle implements to determine its location inreal space; and a controller. The controller can: calculate a nominalpath between the autonomous vehicle's current location and a destinationbased on the navigation map; determine the location of the autonomousvehicle in real space over time based on sensor data collected from thesuite of sensors and the localization map; determine the context of ascene around the autonomous vehicle based on these sensor data; elect afuture action (e.g., a navigational decision) to remain on or deviatefrom the nominal path based on the context of the scene around theautonomous vehicle and the real geospatial location of the autonomousvehicle; and control actuators within the vehicle (e.g., accelerator,brake, and steering actuators) according to elected decisions.

In one implementation, the autonomous vehicle includes one or more 360°LIDAR sensors arranged on the top of the autonomous vehicle, such as ateach of the front and rear of the autonomous vehicle. Each LIDAR sensorcan output one three-dimensional distance map—such as in the form of a3D point cloud representing distances between the LIDAR sensor andexternal surface within the field of view of the LIDAR sensor—perrotation of the LIDAR sensor (i.e., once per scan cycle). The autonomousvehicle can additionally or alternatively include: a set of infraredemitters configured to project structured light into a field near theautonomous vehicle; a set of infrared detectors (e.g., infraredcameras); and a processor configured to transform images output by theinfrared detector(s) into a depth map of the field. The autonomousvehicle can also include one or more color cameras facing outwardly fromthe front, rear, left lateral, and right lateral sides of the autonomousvehicle. For example, each camera can output a video feed containing asequence of digital photographic images (or “frames”), such as at a rateof 20 Hz. Furthermore, the autonomous vehicle can include a set ofinfrared proximity sensors arranged along the perimeter of the base ofthe autonomous vehicle and configured to output signals corresponding toproximity of objects and pedestrians within one meter of the autonomousvehicle.

The autonomous vehicle can also implement one or more local neuralnetworks to process LIDAR feeds (i.e., sequences of LIDAR images), videofeeds (or sequences of color photographic images), and/or other sensordata substantially in real-time in order to localize the autonomousvehicle to a known location and orientation in real space, to interpret(or “perceive”) its surroundings, and to then select and executenavigational actions. For example, a controller integrated into theautonomous vehicle can: pass LIDAR and video feeds into alocalization/perception neural network to detect and characterize staticobjects—such as lane markers, lane reflectors, curbs, road signs,telephone poles, and building facades—near the autonomous vehiclesubstantially in real-time; and then compare types and relativelocations of these static objects to a localization map to determine theautonomous vehicle's position in real space. In this example, the neuralnetwork can also detect and characterize dynamic objects—such as othervehicles, pedestrians, and cyclists—in the LIDAR and video feeds; andthe controller can perceive the autonomous vehicle's local environmentbased on proximity, speed, and types of these nearby dynamic objects.The controller can then select a next navigational action—such asincluding a target wheel angle, road speed, acceleration, ordeceleration (e.g., brake position)—to move toward a specifieddestination based on the autonomous vehicle's current position and thescene around the vehicle, such as by passing the autonomous vehicle'sposition, identified dynamic objects in the local scene, and theautonomous vehicle's destination into a navigational model (e.g., anavigational neural network). The autonomous vehicle can thereforeimplement one or more neural networks locally to determine its location,perceive its surroundings, and select future actions.

The remote computer system can implement Blocks of the first methodS100: to collect LIDAR and color image data from autonomous vehiclesand/or manually-operated road vehicles with similar sensor suites; tocompile these LIDAR and color image data into a visualization that, whenrendered within an annotation portal, enables an human annotator toquickly attribute localization-, perception-, and motionplanning-related labels to these optical data with limited mental load;and to generate and refine localization/perception, navigational, and/orother neural networks—for implementation by these autonomousvehicles—over time based on labels manually inserted into thisvisualization by the human operator.

The first method S100 is described herein as executed by the system toassist a human annotator annotating 3D LIDAR and 2D image data recordedby a ground-based passenger, commercial, or fleet vehicle. However,Blocks of the first method S100 can be executed by the system to assistthe human annotator when annotating 3D LIDAR and 2D image data recordedby a vehicle of any other type.

As described below, the first method S100 can additionally oralternatively be executed by the system to assist a techniciandiagnosing sensors on the autonomous vehicle or locally by theautonomous vehicle to generate a visual representation of its perceptionof its local environment, as described below.

1.3 Vertical Image Planes

To prepare 3D LIDAR and 2D image data recorded by an autonomous vehiclefor manual annotation by a human annotator, the system can: generate avirtual environment including a vehicle origin representing a referencepoint on the autonomous vehicle; and define a LIDAR origin of a LIDARsensor and a camera origin of the color camera relative to the vehicleorigin in the virtual environment based on known positions andorientations of the LIDAR sensor and the camera relative to thereference point on the autonomous vehicle. (The annotation portal canlater inject 3D LIDAR frames—such as in the form of 3D point clouds—intothe virtual environment relative to the LIDAR origin.) The system canthen define an image plane onto which a 2D image recorded by the camerais projected into the virtual environment.

1.3.1 Static Image Plane

In one implementation, the system calculates a ray defining a projectionof the principle axis of the camera onto a horizontal plane—passingthrough the camera origin—in the virtual environment based on a knownposition and orientation of the camera on the autonomous vehicle.

In this implementation, the system then specifies a length of the rayfrom the camera origin. In one example, the system calculates a raylength that places the end of the ray—opposite the camera origin—in thecenter of an adjacent road lane. In this example, for 2D images recordedby a camera arranged at a yaw angle of 45° to the left of thelongitudinal axis of an autonomous vehicle traversing roads in theUnited States in which a 12-foot (or 3.7-meter) lane width is standard,the system can calculate a ray length of 4.3 meters (or 3.7meters/sin(45°)×sin(90°)) for a ray extending 45° to the left of thelongitudinal axis of the autonomous vehicle, which may locate the end ofthe ray at approximately the center of a lane to the immediate left ofthe autonomous vehicle, assuming that the autonomous vehicle isoccupying the approximate center of its lane.

The system can then: define an image plane normal to the rayintersecting the end of the ray (i.e., offset from the camera origin inthe horizontal plane by the ray length); and calculate a transform thatprojects the field of view of the camera onto the image plane based onthe known position and orientation of the camera on the autonomousvehicle (e.g., relative to the reference point on the autonomousvehicle) and based on known intrinsic and extrinsic properties of thecamera (e.g., focal length, imager dimensions, zoom, inherent opticalaberration). The system (e.g., the annotation portal) can later projecta sequence of 2D images—recorded by the camera—onto the image planebased on this transform while simultaneously rendering concurrent LIDARframes in the virtual environment.

The system can repeat this process to define one image plane for eachother color camera on the autonomous vehicle.

Furthermore, the system is described above and below as defining animage plane with its horizontal center virtually centered on acorresponding lane proximal the autonomous vehicle. However, the systemcan align any other horizontal reference point in the image plane withthe center of the corresponding lane. For example, for a color cameraplane of a left-forward camera, the system can align a pixel one-thirdof the width of the image plane from the right edge of the image planewith the center of the corresponding lane.

1.3.2 Multiple Image Planes per Camera

In another implementation, the system defines multiple image planes percamera on the autonomous vehicle. In one example, for a left-forwardcolor camera arranged on the autonomous vehicle (e.g., a camera defininga principle axis angularly offset from the longitudinal axis of theautonomous vehicle by 45°), the system can: insert a vehicle origin intothe virtual environment; insert a LIDAR origin and a camera origin—forthe left-forward color camera—into the virtual environment relative tothe vehicle origin; and calculate a ray defining a projection of theprinciple axis of the camera onto the horizontal plane intersecting thecamera origin in the virtual environment; as described above. The systemcan then: locate a first image plane normal to the ray with thehorizontal center of the first image plane laterally offset from thevehicle origin by a standard or average lane width in the geographicregion occupied by the autonomous vehicle (e.g., 3.7 meters in theUnited States); and calculate a first transform that projects the fieldof view of the left-forward camera onto the image plane based on theknown position, orientation, intrinsic properties, and extrinsicproperties of the left-forward camera. The system can also: locate asecond image plane normal to the ray with the horizontal center of thesecond image plane laterally offset from the vehicle origin by twice thestandard lane width in this geographic region; and calculate a secondtransform that projects the field of view of the left-forward cameraonto the second image plane based on the known position, orientation,intrinsic properties, and extrinsic properties of the left-forwardcamera.

Thus, during annotation of 3D LIDAR and 2D image data recorded by thisautonomous vehicle, the system (e.g., the annotation portal) can applythe first transform to a color image recorded by the left-forward camerato project the color image onto the first image plane and simultaneouslyapply the second transform to the same color image to project the colorimage onto the second image plane while rendering a concurrent 3D LIDARframe within the virtual environment. The system can repeat this processfor a subset of concurrent 2D images and 3D LIDAR frames recorded by theautonomous vehicle according to the first and second transforms.

Therefore, by rendering a sequence of color images recorded by theleft-forward camera onto the first and second image planes according tothis process while simultaneously rendering concurrent LIDAR frameswithin the virtual environment, the system can: depict clusters ofpoints representing a first vehicle—moving relative to the autonomousvehicle in the lane immediately to the left of the autonomous vehicle—passing through the first image plane and intersecting a visual colorrepresentation of the first vehicle in images rendered on the firstcolor image frame; and simultaneously depict clusters of pointsrepresenting a second vehicle—moving relative to the autonomous vehiclein a second lane to the left of the autonomous vehicle—passing throughthe second image plane and intersecting a visual color representation ofthe second vehicle in images rendered on the first color image frame.

The system can implement similar methods and techniques to define oneimage plane approximately centered in each lane depicted in a 2D imagerecorded by the autonomous vehicle. For example, the system can access amap or database of georeferenced lane positions along public roadswithin a geographic region and define image planes—centered on adjacentlanes defined in the map or database—based on geolocation tags stored inmetadata of 3D LIDAR and/or 2D images recorded by the autonomousvehicle.

1.3.3 Derived Image Plane Offset Distances

In another implementation, the system can: implement computer visionand/or artificial intelligence techniques to automatically detect lanemarkers in a 2D image recorded by a camera on the autonomous vehicle;estimate a real lateral distance between parallel lane markers detectedin the 2D image; and define image planes in the virtual environmentbased on this lateral distance between parallel lane markers. Forexample, the system can scan a color image recorded by a forward-facingcamera on the autonomous vehicle for lane markers; and estimate lateraloffset distances from the reference point on the autonomous vehicle tocenters of lanes to the left and right of the autonomous vehicle basedon locations of these lane markers and a known position of theforward-facing camera on the autonomous vehicle. The system can thendefine image planes with centers offset from the vehicle origin in thevirtual environment by these lateral offset distances.

For example, if the system estimates a first lateral offset distance of2.0 meters from the reference point on the autonomous vehicle to a firstlane immediately to the left of the autonomous vehicle and a secondlateral offset distance of 5.5 meters from the reference point on theautonomous vehicle to a second lane to the left of the autonomousvehicle, the system can define a first image plane centered 2.0 metersto the left of the vehicle origin and a second image plane centered 5.5meters to the left of the vehicle origin in the virtual environment fora left-forward-facing camera on the autonomous vehicle. In this example,if the system also estimates a third lateral offset distance of 1.7meters from the reference point on the autonomous vehicle to a thirdlane immediately to the right of the autonomous vehicle, the system candefine a third image plane centered 1.7 meters to the right of thevehicle origin in the virtual environment for a right-forward-facingcamera on the autonomous vehicle. In this example, the system cansimilarly define image planes for left- and right-facing cameras on theautonomous vehicle.

In this implementation, the system can alternatively detect lane markersin the 3D LIDAR frames, estimate lateral distances from the referencepoint on the autonomous vehicle to the centers of adjacent lanes, anddefine image planes based on these LIDAR-based lateral distances.

The system can therefore derive offset distances for image planes in thevirtual environment based on data extracted from 3D LIDAR and/or 2Dimages recorded by the autonomous vehicle. The system can alsorecalculate these offset distances intermittently, for each color imageframe recorded by the autonomous vehicle, or when the system detectslane markers in these optical data (e.g., with at least a minimum degreeof confidence), etc.

1.3.4 Image Plane Offset Distance by Color Image Overlap

In yet another implementation, the system defines ray lengths (orlateral offset distances) for image planes for cameras on the autonomousvehicle such that a 2D image recorded by a first camera and projectedonto the image plane overlaps—by a limited horizontal distance (e.g.,5%)—a color image recorded by a second camera on the autonomous vehicleand projected onto an adjacent color plane.

(The autonomous vehicle can implement similar methods and techniques todefine a longitudinal offset distances for an image plane—that is, aforward distance from a camera, along the camera's field of view, to thecorresponding image plane, as described below.)

1.3.5 Manual Image Plane Offset Distance

In another implementation, the system defines an image plane based on alateral offset distance (or a ray length) set manually by a humanannotator through the annotation portal. For example, the humanannotator can manually adjust the position of an image plane relative tothe virtual origin the virtual environment—such as with aslider—according to the annotation portal's personal preferences oruntil the virtual environment depicts alignment between clusters ofLIDAR points moving through the image plane and object depicted in asequence of 2D images projected onto the image plane.

1.4 Annotation Portal

An annotation portal—executing on a local computer system—can download3D LIDAR and 2D image data, such as from a remote database via acomputer network (e.g., the Internet). The annotation portal can receivedefinitions for image planes in the virtual environment from thecomputer network, or the annotation portal can implement the foregoingmethods and techniques to define these image planes locally.

As shown in FIGS. 3A-3D, the annotation portal can then render thevirtual environment, including a sequence of 3D LIDAR frames andconcurrent sequences of 2D images projected onto corresponding imageplanes, and a predefined list of object labels, such as “car,” “truck,”“motorcycle,” “pedestrian,” “tree,” etc. As the annotation portalrenders these 3D LIDAR and 2D image data (e.g., at a nominal or reducedplayback speed), the human annotator can: manually select a cluster ofLIDAR points (or “pixels”), such as by selecting a representative pointwithin the cluster or by drawing a virtual box around the cluster; andactivate an object label—in the predefined set of object labels—for thisselection in the current LIDAR frame. In this implementation, when thehuman annotator selects a representative point or the entirety of thecluster of LIDAR points, the system can highlight a corresponding regionin a 2D image projected onto an image plane in the virtual environment,such as based on proximity of this cluster of LIDAR points to a discreteobject detected in the projected 2D image as the cluster of LIDAR pointspasses through the image plane. Furthermore, when the human annotatoractivates a label for this cluster of LIDAR points, the annotationportal can indicate this label on the corresponding region of theprojected 2D image, such as by highlighting the corresponding region ofthe projected 2D image in a color assigned to a type of the activelabel, which may better depict a correspondence or correlation betweenconcurrent 3D LIDAR and 2D color image data, reduce mental load, andthus yield fewer annotator mistakes.

Alternatively, as the annotation portal renders these 3D LIDAR and 2Dimage data within the virtual environment, the human annotator can:manually select an object depicted in a 2D image projected onto an imageplane, such as by selecting a representative point within a discreteobject detected in the projected 2D image by the annotation portal (orby the remote computer system) or by drawing a virtual box around thediscrete object; and activate an object label—in the predefined set ofobject labels—for this selection in the projected 2D image. In thisimplementation, when the human annotator selects the discrete object inthe projected 2D image, the system can highlight a corresponding pointor cluster of LIDAR points in the concurrent 3D LIDAR frame in thevirtual environment, such as based on proximity of the discrete regionof the projected 2D image to this cluster of LIDAR points to a discreteobject detected in the projected 2D image as the cluster of LIDAR pointspasses through the image plane. Furthermore, when the human annotatoractivates a label for this discrete object represented in the projected2D image, the annotation portal can indicate this label on thecorresponding cluster of LIDAR points in the 3D LIDAR frame, such as byhighlighting edges of a 3D box around the cluster of LIDAR points in acolor assigned to a type of the active label.

The annotation portal can then implement object-tracking techniques totrack the discrete object indicated by the human annotator in subsequent3D LIDAR frames and 2D images and to project the active label onto theobject in the subsequent frames.

In the variation described above in which the system (or the annotationportal) defines multiple image planes for one camera on the autonomousvehicle, the annotation portal can similarly: enable the human annotatorto label clusters of LIDAR points passing through various image planescorresponding to this camera; and project these labels ontocorresponding discrete objects—depicted in the same 2D image projectedonto these multiple image planes; or vice versa.

1.5 2D Image Segmentation

In the implementations described above in which the system definesmultiple image planes for a camera on the autonomous vehicle, the systemcan: implement computer vision techniques (e.g., edge detection, objectdetection) to detect discrete objects in a 2D image frame recorded bythis camera; segment the image by detected objects; and estimatedistances of these objects from the camera, such as based on sizes ofthese objects and/or changes in overlaps of these objects in precedingframes recorded by the camera. The system can then: associate each imagesegment with a nearest image plane—in the set of image planes definedfor the camera—based on estimated distances of each object in the imageand offset distances for each image plane in the set; project eachsegment of the image onto its associated image plane; and render theseprojections of all segments of the image simultaneously with concurrent3D LIDAR data in the virtual environment.

The system can repeat this process for subsequent 2D images recorded bythe camera as 3D LIDAR and 2D images recorded by the autonomous vehicleare replayed in the annotation portal such that clusters of LIDARpoints—at different distances from the vehicle origin in the virtualenvironment and representing distinct objects in the field around theautonomous vehicle—cross these image planes as segments of color imagesprojected onto these image planes depict the same objects moving acrossthese image planes, thereby enabling the human annotator to quickly,visually link clusters of LIDAR points at different distances from theautonomous vehicle with discrete objects represented in segments of 2Dimages.

1.6 Horizontal Plane

In one variation, the system (e.g., the remote computer system, theannotation portal) implements similar methods and techniques: to project2D images recorded by cameras on the autonomous vehicle onto a groundplane in the virtual environment; and to render concurrent 3D LIDARframes in the virtual environment, thereby enabling the human annotatorto visually link clusters of LIDAR points to discrete objects depictedin a composite 2D color ground plane image in the virtual environment.

In this variation, the annotation portal can also implement methods andtechniques described above to record object labels written to clustersof LIDAR points by the human annotator, to project these object labelsonto objects depicted in the composite 2D color ground plane image,and/or to project these object labels onto corresponding regions ororiginal 2D images recorded by the autonomous vehicle. The annotationportal can additionally or alternatively record object labels written todiscrete objects depicted in the composite 2D color ground plane imageby the human annotator and to project these object labels onto clustersof LIDAR points.

1.7 Alternative Application: Technician Portal

In another implementation, the system accesses 3D LIDAR and 2D imagesrecorded by an autonomous vehicle and implements Blocks of the firstmethod S100 to compile these 3D LIDAR and 2D images into one 3Denvironment within a technician portal. The system can inject 3D LIDARdata into the virtual environment relative to a vehicle originrepresenting a reference point on the autonomous vehicle based on knowndefault positions and orientations of LIDAR sensors—that recorded these3D LIDAR data—relative to this reference point on the autonomousvehicle. By implementing techniques similar to those described above,the system can define image planes relative to the vehicle origin andproject 2D images onto these 2D image planes based on known defaultpositions and orientations of color cameras— that recorded these 2Dimages—relative to the reference point on the autonomous vehicle. Thesystem can render these projected 2D images and concurrent 3D LIDAR datawithin the virtual environment. If the color cameras and LIDAR sensorsremain in their default (or “intended”, “selected”, “prescribed”, or“designed”) positions and orientations on the autonomous vehicle, acluster of LIDAR points representing an object (e.g., another vehicle)may pass through an image plane as a representation of this same objectdepicted in a 2D image is rendered along a similar path along the imageplane; a technician viewing this sequence of 3D LIDAR and 2D images inthe technician portal may therefore visually link the cluster of LIDARpoints and the object in the projected 2D image as representing a commonobject and quickly visually confirm alignment between the LIDARsensor(s) that recorded these 3D LIDAR data and the color camera thatrecorded these 2D images given such alignment between theserepresentations of the object in the virtual environment.

However, if the color cameras and LIDAR sensors have shifted from theirdefault positions and orientations on the autonomous vehicle, a clusterof LIDAR points representing an object (e.g., another vehicle) may beoffset from a representation of the same object depicted in a 2D imageprojected onto the image plane; a technician viewing this sequence of 3DLIDAR and 2D images in the technician portal may therefore quicklydetect misalignment between the LIDAR sensor(s) that recorded these 3DLIDAR data and the color camera that recorded these 2D images given suchoffset between these representations of the same object 3D LIDAR and 2Dimage data rendered in the virtual environment. Accordingly, thetechnician can flag the autonomous vehicle for maintenance or repair.For example, the technician can remotely disable autonomous operation ofthe autonomous vehicle through the technician portal until theautonomous vehicle has been repaired and alignment of sensors on theautonomous vehicle have been re-verified.

In one example, when the autonomous vehicle detects possible occurrenceof a collision (or other high-impact event, such as impact with apothole when traveling at speed), the autonomous vehicle can:automatically pull over into a shoulder and/or median; and stream 3DLIDAR and 2D image data to a remote server as other vehicles pass thestopped autonomous vehicle. The remote server can then serve these datato the technician portal, which can implement Blocks of the first methodS100 to render these 3D LIDAR and 2D images within a virtual environmentwithin the technician portal. By then determining whether passingvehicles depicted in 2D images projected onto image planes in thevirtual environment sufficiently align to like clusters of LIDAR pointsin concurrent LIDAR frames rendered in the virtual environment, thetechnician can quickly visually ascertain whether the collision affectedrelative location sensors on the autonomous vehicle. Accordingly, thetechnician can selectively disable autonomous operation of theautonomous vehicle, permit the autonomous vehicle to autonomouslynavigate (at reduced speed) to a repair facility, or confirm that nodamage is indicated.

The system can therefore implement Blocks of the first method S100 toalign 3D LIDAR and 2D image data within one 3D virtual environment andto render this 3D virtual environment for a technician—with minimalprocessing to prepare these data for rendering—in order to enable thetechnician to quickly visually discern whether sensors in the autonomousvehicle are aligned. For example, the system can present 3D LIDAR and 2Dimage data to a technician through the technician portal, as describedabove: to enable the technician to remotely diagnose the autonomousvehicle, such as following a possible accident or impact; to enable thetechnician to achieve and confirm gross alignment of sensors on theautonomous vehicle when servicing the autonomous vehicle; or to enablethe technician to set and refine relative positions of sensors on theautonomous vehicle during assembly of a sensor suite on the autonomousvehicle.

1.8 Alternative Application: Rider Portal

In yet another implementation, an autonomous vehicle accesses 3D LIDARand 2D images recorded by an autonomous vehicle and implements Blocks ofthe first method S100 to compile these 3D LIDAR and 2D images into one3D environment within a rider portal rendered in real-time on a displaywithin the autonomous vehicle and visible to occupants in the autonomousvehicle (e.g., a “rider portal”). In particular, the autonomous vehiclecan implement Blocks of the first method S100 to: define an image planewithin the virtual environment for each color camera on the autonomousvehicle; inject 3D LIDAR data recorded by LIDAR sensors on theautonomous vehicle into the virtual environment in real-time; andproject color images recorded by the color cameras onto corresponding 2Dimage planes in the virtual environment in real-time. By implementingBlocks of the first method S100 to compile these 3D LIDAR and 2D imagedata into one 3D virtual environment, the autonomous vehicle can depicta representation of the autonomous vehicle's perception of its localenvironment to its occupants in (very near) real-time withoutnecessitating extensive processing power and without producingperceptible latency between recordation and presentation of these datathat may occur when stitching 2D image data into 3D color images andaligning these 3D color images with concurrent 3D LIDAR images.

The autonomous vehicle can therefore implement Blocks of the firstmethod S100 in real-time during autonomous operation in order tovisually communicate its perception of its local environment to anoccupant. The autonomous vehicle can also automatically labelobjects—such as pedestrians and other vehicles—detected in this virtualenvironment in real-time, thereby assuring the occupant that theautonomous vehicle's comprehension of its local environment is corrector sufficiently complete. Furthermore, by projecting 2D image data ontoimage planes in the 3D virtual environment while concurrent 3D LIDARdata is also rendered in the 3D virtual environment, the autonomousvehicle can generate a visual representation of its local environmentthat differs fundamentally in appearance from that of the real-worldenvironment that it represents while also avoiding transformation of 2Ddata into 3D data, thereby reducing opportunity for content rendered bythe autonomous vehicle in the rider portal to nauseate an occupantviewing the rider portal during operation of the autonomous vehicle.

2. Second Method

As shown in FIG. 4 , a second method S200 for augmenting 3D depth mapdata with 2D color image data includes: accessing a first 2D color imagerecorded at a first time via a 2D color camera arranged on an autonomousvehicle in Block S210; accessing a first 3D point cloud recorded atapproximately the first time via a 3D depth sensor arranged on theautonomous vehicle in Block S212, the 3D depth sensor and the 2D colorcamera defining intersecting fields of view and facing outwardly fromthe autonomous vehicle; detecting a first cluster of points in the first3D point cloud representing a first continuous surface approximating afirst plane in Block S220; isolating a first cluster of color pixels inthe first 2D color image depicting the first continuous surface in BlockS230; projecting the first cluster of color pixels onto the first planeto define a first set of synthetic 3D color points in the first 3D pointcloud in Block S232, the first cluster of points and the first set ofsynthetic 3D color points representing the first continuous surface; andrendering points in the first 3D point cloud and the first set ofsynthetic 3D color points on a display in Block S240.

One variation of second method S200 includes: accessing a first 2D colorimage recorded at a first time via a 2D color camera arranged on anautonomous vehicle in Block S21 o; accessing a first 3D point cloudrecorded at approximately the first time via a 3D depth sensor arrangedon the autonomous vehicle, the 3D depth sensor and the 2D color cameradefining intersecting fields of view and facing outwardly from theautonomous vehicle in Block S212; detecting a first cluster of points inthe first 3D point cloud representing a first continuous surfaceapproximating a first plane in Block S220; isolating a first cluster ofcolor pixels in the first 2D color image depicting the first continuoussurface in Block S230; projecting the first cluster of color pixels ontothe first plane to define a first set of synthetic 3D color points inBlock S232; compiling the first cluster of points and the first set ofsynthetic 3D color points, representing the first continuous surface,into a first 3D frame in Block S232; detecting characteristics of anobject, comprising the continuous surface, based on the first cluster ofpoints and the first set of synthetic 3D color points in the first 3Dframe in Block S250; based on characteristics of the object, electing anext navigational action in Block S252; and autonomously executing thenext navigational action in Block S254.

2.1 Applications

Generally, in the second method S200, the autonomous vehicle canimplement methods and techniques similar to the first method S100 toselectively inject 2D pixels—from a 2D color image recorded by theautonomous vehicle—into a concurrent 3D point cloud in order to increasedensity of visual information depicting select objects or surfaces inthe 3D point cloud, as shown in FIGS. 5 and 6 . In particular, theautonomous vehicle can: interpret a surface represented by a cluster ofpoints in a 3D point cloud (e.g., a depth map, a 3D LIDAR image);isolate a segment of a concurrent 2D color image that depicts thissurface; and inject color pixels from this segment of the 2D color imageonto this surface in the 3D point cloud in order to increase density ofpoints depicting this surface in the 3D point cloud. The autonomousvehicle can then render this augmented 3D point cloud on an interiordisplay. The combination of original 3D points and “synthetic 3D colorpoints” thus incorporated into the augmented 3D point cloud rendered onthis interior display may enable a rider occupying the autonomousvehicle to better visualize the environment and objects (e.g., othervehicles, pedestrians) around the autonomous vehicle, enable the riderto better comprehend the autonomous vehicle's perception of itsenvironment, and thus improve the rider's comfort and confidence in theautonomous vehicle.

2.1.1 Augmented 3D Point Cloud for Rider

Generally, a 3D point cloud output by a LIDAR sensor or other 3D depthsensor on the autonomous vehicle may contain sparse representations ofobjects in the field around the autonomous vehicle or otherwise containlimited densities of (e.g., high gap distances between) pointscorresponding to discrete objects in this field, which may result indifficulty for humans to interpret types and other characteristics ofthese objects based exclusively on such point-based representations in a3D point cloud. Therefore, in one example, the autonomous vehicle canimplement Blocks of the second method S200 to selectively augment acluster of points—in a 3D point cloud—depicting a surface on anothervehicle in or near the autonomous vehicle's path with color pixels froma region of the concurrent 2D color image depicting this same surface,thereby “filling” gaps between original depth map points in the 3D pointcloud with “synthetic 3D color points.” By thus leveraging datacontained in a concurrent 2D color image to reduce gap distance betweenpoints depicting this vehicle in the 3D point cloud and rendering thisaugmented 3D point cloud on an interior display for a rider, theautonomous vehicle can thus enable the rider to immediately interpretthese points as representing another vehicle and maintain confidencethat the autonomous vehicle perceives this other vehicle and will avoidcollision with this other vehicle.

The autonomous vehicle can similarly augment clusters of points in the3D point cloud depicting other objects with synthetic 3D color pointsderived from color pixels in corresponding regions of the concurrent 2Dcolor image. However, by detecting particular objects of interest (e.g.,other vehicles, pedestrians, road signs, lane markers) in the fieldaround the autonomous vehicle, selectively augmenting clusters of pointsin the 3D point cloud representing these objects of interest, and thenrendering this augmented 3D point cloud on an interior display, theautonomous vehicle can limit computational load required to merge this3D point cloud and the 2D color image while also ensuring that objects(likely) of greatest interest to a rider are presented at pointdensities (more) sufficient for the rider to immediately comprehend.Therefore, the autonomous vehicle can execute Blocks of the secondmethod S200 to selectively merge 3D point cloud and 2D color image datainto a 3D representation of the field around the autonomous vehicle withobjects depicted at point densities approximately proportional toimportance of these objects to a rider inside the autonomous vehicle.

2.1.2 Augmented 3D Point Cloud for Remote Operator and Human Annotator

The autonomous vehicle (or a remote computer system) can implementsimilar methods and techniques to selectively merge 3D point cloud and2D color image data into a 3D representation of the field around theautonomous vehicle with objects depicted at point densitiesapproximately proportional to importance of these objects to a remoteoperate upon assuming remote manual control of the autonomous vehicle,such as in response to a disengagement event at the autonomous vehicle.Similarly, the autonomous vehicle (or the remote computer system) canimplement these methods and techniques to selectively merge 3D pointcloud and 2D color image data into an augmented 3D point cloud thatenables a human annotator to more easily comprehend characteristics ofobjects depicted in the original 3D point cloud and to label theseobjects in less time and/or with greater accuracy.

2.1.3 Augmented 3D Point Cloud for Autonomous Vehicle Perception

The autonomous vehicle can: implement similar methods and techniques toselectively merge 3D point cloud data (e.g., from a LIDAR or RADARsensor, including 3D position, intensity, and/or velocity information)and 2D color image data into an augmented 3D point cloud; and then passthis augmented 3D point cloud into a perception pipeline (e.g., a deeplearning or artificial intelligence network) to perceive types,trajectories, and/or other characteristics of objects, as shown in FIG.4 . In particular, by filling gaps between original depth map pointsdepicting a particular object in the 3D point cloud with synthetic 3Dcolor points derived from color pixels depicting this same object in theconcurrent 2D color image, the autonomous vehicle can process theaugmented 3D point cloud to detect, identify, and characterize theparticular object with higher confidence due to higher density ofinformation for this particular object in the augmented 3D point cloud.(By processing this augmented 3D point cloud that includes both original3D points and synthetic 3D color points, the autonomous vehicle can alsoavoid both separate analyses of the 2D color image and subsequentcombination of separate analyses of the original 3D point cloud and 2Dcolor image, thereby reducing processing time and/or computationalload.)

2.1.4 Plane-Based Augmentation

Furthermore, the autonomous vehicle can selectively augment clusters ofpoints—falling on or (very) near a plane—with synthetic 3D color points.In particular, deriving a 3D surface spanning a cluster of pixels in a3D point cloud and then projecting 2D color pixels from a 2D color imageonto this 3D surface may be both computationally intensive and prone toerror due to different dimensions of these data and differentperspectives of the 3D depth sensor and 2D color camera. Therefore, theautonomous vehicle can: identify a cluster of points that fall on orvery near planes in the 3D point cloud; and then selectively augmentthis cluster of points with 2D data by porting 2D color pixels from the2D color image onto a 2D (i.e., planar) manifold that (approximately)intersects this cluster of points in the 3D point cloud, therebypreserving 2D color pixel data within a 2D domain, reducing oreliminating need for interpreting complex 3D surfaces in the 3D pointcloud, and reducing opportunity for error in position of color datainjected into the 3D point cloud. The autonomous vehicle can alsoimplement a simple (e.g., homogenous) transformation matrix to port 2Dcolor pixels into the 3D point cloud—such as based on a known orcalibrated offset between the 3D depth sensor and the 2D colorcamera—thereby limiting computational load necessary to merge thesedata.

Additionally or alternatively, the autonomous vehicle can implementBlocks of the second method S200 to augment clusters of points depictingstatic objects in a 3D point cloud with data from a concurrent 2D colorimage, such as selectively for static objects containing iconography orother content of interest, including billboards, road signs, licenseplates, and building facades.

The second method S200 is described herein as executed in real-time bythe autonomous vehicle to fuse data from a 2D color image with aconcurrent 3D point cloud recorded by sensors on the autonomous vehicle.However, the autonomous vehicle can also repeat this process regularlyover time, such as at a rate of 20 Hz or once per 3D point cloudgenerated by a 3D depth sensor. The autonomous vehicle can also executeBlocks of the second method S200 to fuse color data from multiple 2Dcolor images with a single 3D depth map. The autonomous vehicle can alsocombine multiple concurrent 3D depth maps generated by multiple 3D depthsensors on the autonomous vehicle into one composite 3D depth map andthen fuse color data from one or more concurrent 2D color images withthis composite 3D depth map according to the second method S200.Additionally or alternatively, a remote computer system can access (raw)2D color images and 3D point clouds recorded previously by an autonomousvehicle and then implement Blocks of the second method S200 to fusethese data, such as in preparation for remote manual or automatedannotation.

2.2 Data Collection

Blocks S210 and S220 of second method S200 recite: accessing a first 2Dcolor image recorded at a first time via a 2D color camera arranged onan autonomous vehicle; and accessing a first 3D point cloud recorded atapproximately the first time via a 3D depth sensor arranged on theautonomous vehicle in Block S212, the 3D depth sensor and the 2D colorcamera defining intersecting fields of view and facing outwardly fromthe autonomous vehicle. Generally, in Blocks S210 and S212, theautonomous vehicle can access color and 3D depth data recorded—atsimilar times—by sensors on the autonomous vehicle.

In one implementation, the autonomous vehicle includes a set of spinningLIDAR sensors (i.e., a “3D depth sensor) arranged on the autonomousvehicle, such as one LIDAR sensor mounted at each corner of theautonomous vehicle or a set of LIDAR sensors integrated into a roof rackmounted to the roof of the autonomous vehicle. Each LIDAR sensor canoutput one 3D LIDAR image—such as in the form of a “3D point cloud”representing distances between the LIDAR sensor and external surfaceswithin the field of view of the LIDAR sensor—per rotation of the LIDARsensor (i.e., once per scan cycle). For example, each LIDAR sensor caninclude a stack of pixels (e.g., 16 or 32 pixels arranged in a linearvertical array) mounted on a motorized rotary table; during a scancycle, the LIDAR sensor can rotate the stack of pixels and record adistance value from each pixel in the stack at each of many (e.g., 1000or 4500) points per rotation to produce a LIDAR image containing a 3Dpoint cloud characterized by a relatively high azimuthal resolution(e.g., 0.3° or 0.08°) and a lower vertical resolution (e.g., 1.7° or0.8°). Furthermore, because the LIDAR sensor sweeps a column of pixelsabout its rotational axis, the LIDAR sensor may represent an object nearthe autonomous vehicle and in the field of view of the LIDAR sensor asrows of points in a LIDAR image. In particular, a large object nearerthe autonomous vehicle may be represented by a larger number of rows,each containing a larger number of points, in this LIDAR image; and asmaller object further from the autonomous vehicle may be represented bya smaller number of rows—each containing fewer points—in this LIDARimage.

The autonomous vehicle can also include a suite of 2D color cameras,such as grayscale or RGB cameras: arranged on the roof of the autonomousvehicle; located in multiple orientations about the autonomous vehicle(e.g., facing the front, front corners, sides, and rear of theautonomous vehicle); and each defining a field of view that overlaps thefield of view of at least one LIDAR sensor on the autonomous vehicle.

During operation, 3D depth sensors on the autonomous vehicle canregularly output 3D point clouds, such as at a rate of 10 Hz. Cameras onthe autonomous vehicle can similarly regularly output 2D color images,such as at a rate of 24 Hz. During a scan cycle, the autonomous vehiclecan thus: select a last 3D point cloud output by a 3D depth sensor inBlock S210; and select a 2D color image recently output by acamera—defining a field of view intersecting the field of view of the 3Ddepth sensor—at a time nearest the record time of the 3D point cloud inBlock S212.

However, the autonomous vehicle can include one or more 3D depth sensorsand 2D color cameras of any other type or configuration.

2.3 Object Detection and Perception

The autonomous vehicle can then implement baseline perception and/ordeep learning perception techniques to detect objects in the fieldaround the autonomous vehicle from the 3D point cloud.

In one implementation, the autonomous vehicle executes a baselineperception pipeline to derive object perception data from the 3D pointcloud, such as including predictions of types and relative velocities ofobjects represented by points in the 3D point cloud. The autonomousvehicle concurrently implements a deep learning pipeline to derivehigher-resolution object perception data from the 3D point cloud, suchas including classification of objects and predictions for dimensions(e.g., lengths) of objects depicted at lower point densities orpartially obscured in the 3D point cloud. Therefore, in thisimplementation, the autonomous vehicle implements both: baselineperception techniques to determine whether a cluster of points in the 3Dpoint cloud represents a vehicle element or another type of object(e.g., a “soft classification” to predict whether this cluster of LIDARpoints represents a vehicle element); and deep learning perceptiontechniques to predict a type and other more specific characteristics ofa vehicle element (e.g., regression techniques to predict a type of thevehicle, a length of the vehicle, and a vehicle dynamics model mostrepresentative of the vehicle element).

For example, the autonomous vehicle can implement baseline perception topredict whether a cluster of points in a 3D point cloud recorded duringa current scan cycle represents a road vehicle, a pedestrian, a roadsign, a shrub or tree, a building façade, a trashcan, or other discreteobject. Upon determining that a cluster of points corresponds to onediscrete object, the autonomous vehicle can link (or “group”) thesepoints and implement 3D object tracking techniques to track this clusterof points over subsequent 3D point clouds recorded by the 3D depthsensor. The autonomous vehicle can also implement deep learningperception in parallel with baseline perception to verify objectsdetected by the baseline perception pipeline and to derive additionalcharacteristics of the object, such as an orientation, type, and motioncharacteristics of the object. The autonomous vehicle can then select orgenerate a motion model for the object based on characteristicsextracted by the deep learning pipeline and then estimate a trajectoryand/or predict future locations of the object based on this motionmodel.

By tracking clusters of points corresponding to discrete objects in thefield around the autonomous vehicle, the autonomous vehicle can alsodistinguish clusters of points that depict static objects (e.g., roadsigns, lane markers, buildings, telephone poles, curb, trashcans, trees,shrubs) from clusters of points that depict dynamic objects (othervehicles, motorcycles, pedestrians, cyclists). (Additionally oralternatively, the autonomous vehicle can match clusters of points inthe 3D point cloud to static objects labeled in a localization map toidentify immutable objects in the field and to distinguish mutableobjects—and therefore possibly dynamic objects—from these immutableobjects.) The autonomous vehicle can then label or distinguish clustersof points in the 3D point cloud that correspond to dynamic objects fromclusters of points in the 3D point cloud that correspond to staticobjects. The autonomous vehicle can then selectively: augment clustersof points that correspond to dynamic objects with data from theconcurrent 2D color image; and augment clusters of points thatcorrespond to static objects with data from the preceding 3D pointclouds.

However, the autonomous vehicle can implement any other method ortechnique to detect and track clusters of pixels representing discreteobjects in the current 3D point cloud during the current scan cycle.

The autonomous vehicle can then: implement a local motion planner toelect a next navigational action based on perception and characteristicsof these objects in the field around the autonomous vehicle;autonomously execute this next navigational action; and repeat thisprocess for 3D point clouds recorded during each subsequent scan cycleat the autonomous vehicle as the autonomous vehicle autonomouslynavigates to its assigned destination.

2.4 Plane Detection and Surface Selection

Block S220 of second method S200 recites detecting a first cluster ofpoints in the first 3D point cloud representing a first continuoussurface approximating a first plane. Generally, after recording the 3Dpoint cloud during the current scan cycle, the autonomous vehicle canimplement plane detection or plane fitting techniques in order toisolate clusters of points that fall on (or very near) a plane and maytherefore represent an approximately-planar surface in the field of viewof the 3D depth sensor during this scan cycle.

In one implementation shown in FIG. 4 , the autonomous vehicle isolatesmultiple groups of points in the 3D point cloud that correspond todynamic (i.e., “moving”) objects in the field around the autonomousvehicle, as described above. For each group of points in this set, theautonomous vehicle can then: de-noise this group of points to discardoutliers; and implement plane detection or plane fitting techniques tocalculate best-fit planes for subsets (of minimum size) of points inthis filtered group. More specifically, for a filtered group of points,the autonomous vehicle can: calculate a set of planes that each span thelargest quantity of points with the smallest error in this group; andthen cluster subsets of points in this group that fall approximately onthe same planes. For example, for a group of points representing asemi-trailer located to the right and ahead of the autonomous vehicle,the autonomous vehicle can calculate: a first plane that spans a firstcluster of points—in this group—representing the left slab side of thesemi-trailer; and a second plane that spans a second cluster ofpoints—in this same group of points—representing the rear of thesemi-trailer. The autonomous vehicle can implement similar methods andtechniques to calculate planes that approximately intersect clusters ofpoints that represent faces of other dynamic objects in the field, suchas sides of passenger vehicles or torsos of pedestrians.

The autonomous vehicle can implement similar methods and techniques tocalculate planes for clusters of points depicting surfaces of staticobjects in the 3D point cloud, such as clusters of points depicting roadsigns, building facades, sides of parked vehicles, and the road surfacein front of the autonomous vehicle (e.g., up to a maximum distance of 20meters from the autonomous vehicle).

In this implementation, the autonomous vehicle can also set error limitsfor calculating best-fit planes for points in an object. For example,the autonomous vehicle can set higher permissible error limits forsmaller objects and/or for objects characterized by lower pointdensities in the 3D point cloud; and vice versa. In this example, theautonomous vehicle can thus: calculate a first plane for a first clusterof points representing the front of a pedestrian's torso with highpermitted error; calculate a second plane for a second cluster of pointsrepresenting a rear of a passenger vehicle with moderate permittederror; and calculate a third plane for a third cluster of pointsrepresenting a large slab side of a semi-trailer with low permittederror in the current 3D point cloud. The autonomous vehicle can then:discard planes that exhibit error greater than their permissible errorlimits; and selectively augment planes—in the 3D point cloud—thatexhibit less than their permissible error limits with color data fromthe concurrent 2D color image.

The autonomous vehicle can therefore calculate or otherwise identifymultiple (or many) planes that approximate clusters of points in the 3Dpoint cloud, wherein each plane and point cluster pair depicts a surfaceof an object in the field around the autonomous vehicle. However, theautonomous vehicle can implement any other method or technique tocalculate or identify these planes.

2.5 Color Pixel Isolation

Block S230 of second method S200 recites isolating a first cluster ofcolor pixels in the first 2D color image depicting the first continuoussurface; and Block S232 of the second method S200 recites projecting thefirst cluster of color pixels onto the first plane to define a first setof synthetic 3D color points in the first 3D point cloud. Generally, theautonomous vehicle can select a particular plane and correspondingcluster of points in the 3D point cloud to augment with color data fromthe concurrent 2D color image, such as based on a rank or priority forthe plane, cluster of points, or corresponding object as describedbelow. The autonomous vehicle can then: retrieve color data from thecorresponding region of the concurrent 2D color image in Block S230; andproject this color data into the 3D point cloud to generate an“augmented 3D point cloud” in Block S232.

In one implementation, the autonomous vehicle: calculates a 2D boundaryon the plane that encompasses this cluster of points in the 3D pointcloud. The autonomous vehicle can then leverage a known position of thecolor camera relative to the 3D depth sensor—such as stored in analignment transform recently calculated by the autonomous vehicle—toproject the 2D boundary around this cluster of pixels onto theconcurrent 2D color image. The autonomous vehicle can then project colorpixels contained within this projected boundary in the 2D color imageonto the plane in the 3D point cloud and within the 2D boundary thusdefined on this plane. The autonomous vehicle can store these projectedcolor pixels as synthetic 3D color points in the 3D point cloud (now the“augmented 3D point cloud”).

In a similar implementation, after projecting the 2D boundary aroundthis cluster of pixels in the 3D point cloud onto the concurrent 2Dcolor image, the autonomous vehicle can isolate a true boundary of asurface corresponding to this cluster of points by implementing edgedetection or other computer vision techniques to detect an edge nearestthe projected 2D boundary. The autonomous vehicle can then project colorpixels contained from within this true boundary in the 2D color imageback onto the plane in the 3D point cloud—such as based on an inverse ofthe alignment transform to define a set of synthetic 3D color points inthe 3D point cloud. The autonomous vehicle can then store theseprojected color pixels as synthetic 3D color points in the 3D pointcloud.

(In one variation, rather than import color pixels from the 2D colorimage into the 3D point cloud, the autonomous vehicle can: initialize a3D frame with points from the current 3D point cloud, such as ade-noised subset of points from the original 3D point cloud; and augmentthis 3D frame with a set of synthetic 3D color points depicting thefirst continuous surface on the dynamic object. The autonomous vehiclecan then process and/or render this 3D frame—rather than the augmented3D point cloud—as described below.)

However, the autonomous vehicle can implement any other method ortechnique to isolate color pixels in the 2D color image that correspondto the surface represented by the cluster of pixels and approximately bythe corresponding plane in the 3D point cloud in Block S230 and to portthese color pixels into the 3D point cloud accordingly in Block S232.

2.5.1 Multiple Augmented Surfaces per Object

The autonomous vehicle can implement similar methods and techniques toaugment clusters of points—representing different approximately-planarsurfaces on one object—in the 3D point cloud with color data fromcorresponding regions of the concurrent 2D color image, as shown inFIGS. 4 and 6 .

For example, the autonomous vehicle can: select a first cluster ofpoints —in the 3D point cloud—representing a first continuous surfacecorresponding to a first side of an object (e.g., a side of asemi-trailer) proximal the path of the autonomous vehicle in Block S220;calculate a first plane that approximately intersects the first clusterof points with minimum error; isolate a first cluster of color pixels inthe first 2D color image depicting the first continuous surface on theobject in Block S230; and then project the first cluster of color pixelsonto the first plane to define a first set of synthetic 3D colorpoints—in the 3D point cloud—that cooperate with the first set ofsynthetic 3D color points to depict the first continuous surface on theobject in Block S232. Simultaneously, the autonomous vehicle can:identify a second cluster of points—in the 3D point cloud—representing asecond continuous surface corresponding to a second side of the object(e.g., a rear of the semi-trailer) in Block S220; calculate a secondplane that approximately intersects the second cluster of points withminimum error; isolate a second cluster of color pixels in the second 2Dcolor image depicting the second continuous surface on the object inBlock S230; and then project the second cluster of color pixels onto thesecond plane to define a second set of synthetic 3D color points—in the3D point cloud—that cooperate with the second set of synthetic 3D colorpoints to depict the second continuous surface on the object in BlockS232.

The autonomous vehicle can thus augment multiple discontinuous surfaceson the same object represented by points in the 3D point cloud withcolor data from the concurrent 2D color image.

Alternatively, the autonomous vehicle can prioritize augmenting onesurface on an object, such as a largest, approximately-planar surface onthe object depicted in the 3D point cloud. For example, the autonomousvehicle can: isolate a constellation of points depicting a dynamicobject in the current 3D point cloud; interpret a set of surfaces on thedynamic object represented by the constellation of points; and select acluster of points—in this constellation of points—that span or define alargest approximately-planar surface, in the set of surfaces, on thedynamic object.

2.6 Object Filling Prioritization:

The autonomous vehicle can similarly execute Blocks of the second methodS200 to augment surfaces on multiple distinct objects represented bypoints in the 3D point cloud with color data from the concurrent 2Dcolor image, as shown in FIG. 4 . For example, the autonomous vehiclecan augment multiple clusters of points—in one 3D pointcloud—representing surfaces on various discrete vehicles, pedestrians,road signs, and/or building façades with color data from the current 2Dcolor image (or with color data from multiple concurrent 2D color imagesrecorded by a set of color cameras arranged on the autonomous vehicle).

However, the autonomous vehicle can also prioritize objects depicted inthe 3D point cloud and then selectively augment corresponding clustersof points with color data from the concurrent 2D color image.

2.6.1 Dynamic Objects

In one implementation, the autonomous vehicle selectively augmentsclusters of points representing surfaces on dynamic (i.e., moving)objects in the field near the autonomous vehicle, shown in FIGS. 4 and 6.

For example, the autonomous vehicle can implement 3D object trackingtechniques to detect a dynamic object (e.g., another vehicle, apedestrian)—in the field near the autonomous vehicle—located near theautonomous vehicle's current path and to detect a static object in thefield in a sequence of 3D point clouds recorded by the 3D depth sensor.The autonomous vehicle can then: prioritize augmentation of the dynamicobject in the current 3D point cloud with color data from the concurrent2D color image—over augmentation of the static object—because the staticobject may be augmentable (with less computational load) with 3D datafrom past 3D point cloud data; and select the dynamic object forselective augmentation with 2D color data accordingly. The autonomousvehicle can then: select a cluster of points, in the current 3D pointcloud, representing an approximately-planar continuous surface on thedynamic object in Block S220; isolate a cluster of color pixels in theconcurrent 2D color image depicting this continuous surface on thedynamic object in Block S230; and then project the cluster of colorpixels into the current 3D point cloud, thereby augmenting the current3D point cloud with a set of synthetic 3D color points depicting thecontinuous surface on the dynamic object.

Therefore, in this implementation, the autonomous vehicle can prioritizeaugmentation of clusters of points representing other vehicles,pedestrians, and cyclists over road signs, buildings facades, and otherstatic or immutable objects near the autonomous vehicle.

2.6.2 Objects Near Path of Autonomous Vehicle

In a similar implementation, the autonomous vehicle can prioritizeaugmentation of clusters of points depicting objects—in the 3D pointcloud—near the autonomous vehicle's path with color data from theconcurrent 2D color image.

For example, the autonomous vehicle can implement 3D object trackingtechniques to detect a first dynamic object—in the field near theautonomous vehicle—moving toward the autonomous vehicle's path and todetect a second dynamic object—in the field—located remotely from theautonomous vehicle's path in a sequence of 3D point clouds recorded bythe 3D depth sensor. The autonomous vehicle can then prioritizeaugmentation of the first dynamic object in the current 3D point cloudwith color data from the concurrent 2D color image—over augmentation ofthe second dynamic object—such as to accommodate a rider viewing thisaugmented 3D point cloud who may be more interested in or concerned withthe autonomous vehicle's perception of objects in the path of theautonomous vehicle than objects remote from the autonomous vehicle'spath. The autonomous vehicle can then implement methods and techniquesdescribed above to selectively augment a cluster of points—representinga surface on the first dynamic object—in the 3D point cloud with colordata from the concurrent 2D color image.

The autonomous vehicle can implement similar methods and techniques toprioritize augmentation of: objects at closer distances to theautonomous vehicle; objects moving at greater absolute speeds; objectsmoving at greater speeds relative to the autonomous vehicle; or objectsmoving toward the autonomous vehicle or the autonomous vehicle's path.

The autonomous vehicle can similarly prioritize augmentation of smallerobjects over larger objects. For example, because smaller objects may bedepicted with fewer total points in the original 3D point cloud, thesesmaller objects may be more difficult for a rider in the autonomousvehicle to perceive when this 3D point cloud is rendered on the interiordisplay of the autonomous vehicle. (This smaller number of points in the3D point cloud also corresponds to less available data for theautonomous vehicle to perceive this smaller object and may thereforeresult in the autonomous vehicle identifying this object with lessconfidence.) Therefore, the autonomous vehicle can selectively augmentclusters of points—depicting small objects, such as pedestrians andtrashcans, in the 3D point cloud—with color data from the concurrent 3Dpoint cloud.

2.6.3 Point Density

Similarly, the autonomous vehicle can prioritize augmentation ofclusters of points with lower point densities. As in the foregoingexample, objects represented in the 3D point cloud with lower densitiesof points may be more difficult for a rider in the autonomous vehicle toperceive (and yield lower-confidence perception by the autonomousvehicle); the autonomous vehicle can therefore selectively augment theseclusters of points in the 3D point cloud with color data from theconcurrent 2D color image.

In one implementation, that autonomous vehicle identifies a set ofclusters of points in the current 3D point cloud, wherein each clusterof points in the set of clusters of points represents a discrete surfacein the field around the autonomous vehicle and contains points that fallapproximately on a plane. The autonomous vehicle then selects aparticular cluster of points—in this set of clusters of points—that ischaracterized by a density of points less than a threshold point densityand then augments this particular cluster of points with color data fromthe concurrent 2D color image. In this example, the autonomous vehiclecan: extract a distance from the autonomous vehicle to the surfacerepresented by this particular cluster of points in the 3D point cloud;and then calculate the threshold point density inversely proportional tothis distance from the autonomous vehicle to the first continuoussurface. Alternatively, the autonomous vehicle can implement a fixedthreshold point density for all objects in the field and thenselectively augment clusters of points—representing surfaces on dynamicobjects in the field—that exhibit less than this threshold pointdensity.

However, the autonomous vehicle can implement any other schema to selectparticular clusters of points in the 3D point cloud for augmentationwith color data for the concurrent 2D color image.

2.7 Static Object Augmentation With Past 3D Depth Data

In the foregoing implementations, the autonomous vehicle can thereforeaugment a cluster of points representing a dynamic object in the current3D point cloud with synthetic color pixels generated from 2D colorpixels depicting the dynamic object in the concurrent 2D color image,thereby increasing density of points depicting this dynamic object inthe augmented 3D point cloud. In one variation of the method shown inFIGS. 4 and 5 , the autonomous vehicle can also aggregate 3D pointsdepicting a static object—across multiple consecutive and preceding 3Dpoint clouds—and inject these aggregate points into the current 3D pointcloud in Block S260 in order to achieve greater density of pointsdepicting the static object in the 3D point cloud but with lesscomputational load than merging color and depth data for the dynamicobject.

In particular, the autonomous vehicle can: record a first 3D point cloudat a first time; isolate a first set of points representing a staticobject in the first 3D point cloud; record a second 3D point cloud at asecond time; track a change in position of the autonomous vehiclerelative to the static object from the first time to the second time;and then leverage identification of the object as static to project thefirst set of points representing the static object directly into thesecond 3D point cloud based on the known change in position of theautonomous vehicle from the first time to the second time. Theautonomous vehicle can thus directly combine points depicting thisstatic object over multiple 3D point clouds without converting thesepoints between 2D and 3D domains or otherwise processing these points,thereby limiting computational load required to increase density ofpoints depicting this static object in the later 3D point cloud. Theautonomous vehicle can repeat this process over multiple scan cycles toinject points from multiple previous 3D point clouds into the current 3Dpoint cloud such that the current 3D point cloud contains a very highdensity of points depicting static objects in the field nearby.

For example, the autonomous vehicle can: detect a static object in apreceding sequence of 3D point clouds; isolate a second cluster ofpoints—in the current 3D point cloud—that represents this static object;aggregate groups of static points representing the static object in thepreceding sequence of 3D point clouds; and project these groups ofstatic points into the current 3D point cloud based on changes inabsolute position of the autonomous vehicle between the time that thecurrent 3D point cloud was recorded and times that these preceding 3Dpoint clouds were recorded by the autonomous vehicle.

The autonomous vehicle can thus: augment clusters of points representingdynamic objects in the 3D point cloud with synthetic 3D color pointsderived from the concurrent 2D color image; and augment clusters ofpoints representing static objects in the 3D point cloud with pointsderived from the preceding 3D point clouds.

2.8 3D Point Recoloring from Synthetic 3D Color Points

In one variation shown in FIG. 4 , the autonomous vehicle can also colorthe cluster of points with color values of adjacent synthetic 3D colorpoints projected into the 3D point cloud. In one implementation, oncethe autonomous vehicle populates a plane approximating a cluster ofpoints in the 3D point cloud with a set of synthetic 3D color points asdescribed above, the autonomous vehicle can interpolate color values ofpoints in the cluster based on color values stored in these synthetic 3Dcolor points. For example, for each original 3D point in the cluster,the autonomous vehicle can: write a color value from the nearestsynthetic 3D color point to the original 3D point; write an averagecolor value of nearest synthetic 3D color points (e.g., the threenearest synthetic 3D color points) to the original 3D point; linearlyinterpolate a color value of the original 3D point from a short row orcolumn synthetic 3D color points on the corresponding plane nearby; orradially interpolate a color value of the original 3D point fromsynthetic 3D color points surrounding the point.

The autonomous vehicle can therefore selectively color a cluster ofpoints—depicting an approximately-planar surface on a dynamic object inthe 3D point cloud—based on color data derived from the concurrent 2Dcolor image. The autonomous vehicle can implement similar methods andtechniques to selectively augment and color a cluster ofpoints—depicting an approximately-planar surface on a static object ofinterest in the 3D point cloud (e.g., a billboard, road sign, a buildingfaçade)—based on color data derived from the concurrent 2D color image.

The autonomous vehicle can then render this 3D point cloud containingboth: clusters of points depicting dynamic (and static) objects ofinterest shown in color and augmented with synthetic 3D color points;and with monochromatic points depicting other lower-priority (dynamicand static) objects and surfaces in the field.

2.9 3D Point Recoloring from 2D Color Pixels

Additionally or alternatively, the autonomous vehicle can port colorvalues directly from color pixels in the 2D color image onto points inthe 3D point cloud in Block S270 by deriving correspondence between apoint in the 3D point cloud and a pixel in the first 2D color image andthen interpolating a color value of the point directly based on thecolor values stored in the color pixels, as shown in FIGS. 4 and 5 . Forexample, the autonomous vehicle can access an alignment transform forthe color camera and 3D depth sensor and virtually locate the 2D colorimage relative to the 3D point cloud based on the alignment transformand as described above. For each point in the 3D point cloud, theautonomous vehicle can then: implement ray tracing techniques to projecta ray from the origin of the 3D point cloud through the point, asdescribed above; isolate a cluster of pixels in the 2D color image thatfall nearest the ray; implement linear or radial interpolationtechniques to derive a color value of the point from this cluster ofpixels; and then write this color value to the point. The autonomousvehicle can repeat this process to assign color values to all otherpoints in the 3D point cloud. Alternatively, the autonomous vehicle canimplement this process to selectively assign color values to otherpoints representing objects of interest in the 3D point cloud. Yetalternatively, the autonomous vehicle can: implement this process toassign color values to a first subset of points in the 3D point cloud,such as 10% of points uniformly distributed throughout the 3D pointcloud; and then interpolate color values of other points in the 3D pointcloud from this first subset of colored points.

In a similar implementation, the autonomous vehicle: selectively colorspoints in clusters augmented with synthetic 3D color points, asdescribed above; and selectively ports color values directly from colorpixels in the 2D color image onto remaining points in the 3D pointcloud. In particular, coloring points in an augmented cluster withsynthetic 3D color points may result in fewer aberrations or errorsbetween color values mapped to the original depth map points in the 3Dpoint cloud and color values of adjacent synthetic 3D color pointsinjected into the 3D point cloud but may require relatively greatercomputational load per pixel to complete this process. The autonomousvehicle can also color other clusters of points in the 3D point cloudnot located near synthetic 3D color points by directly porting colorvalues from the 2D color image onto these other clusters of points inorder to achieve a full-color 3D point cloud with relatively lesscomputational load per pixel.

However, the autonomous vehicle can implement any other method ortechnique to port color data from the 2D color image into the 3D pointcloud.

2.10 Variation: Color Space to 3D Space

In one variation, rather that identify objects for augmentation in the3D point cloud, the autonomous vehicle initially isolates clusters ofcolor pixels depicting discrete objects in the concurrent 2D color imageand then projects these pixels into the 3D point cloud. For example, theautonomous vehicle can: segment the 2D color image to isolate clustersof color pixels that depict discrete objects; derive boundaries aroundthese objects in 2D color space; select an object in the 2D color image(e.g., based on priority, such as described above); project the boundaryof the object from 2D color space into the 3D point cloud based on astored alignment transform for the 2D color camera and the 3D depthsensor; isolate a group of points bounded by this projected boundary;calculate a set of (i.e., one or more) planes that approximate clustersof points in this group; and then implement the foregoing methods andtechniques to augment each of these clusters of points with synthetic 3Dcolor points based on color pixels that depict this object in the 2Dcolor image.

2.11 Variation: 3D Surface in 3D Point Cloud

As described above, the autonomous vehicle can isolate a set of planesthat approximate surfaces—such as down to a minimum area—of an objectdepicted in a group of points in the 3D point cloud and then selectivelyaugment these planes in the 3D point cloud with color data from theconcurrent 2D color image. In particular, the autonomous vehicle canaugment many planar facets—that approximate an object in the 3D pointcloud—with synthetic 3D color points based on the 2D color image; thesesynthetic 3D color points may thus approximate a non-planar or compoundsurface on the object.

Alternatively, the autonomous vehicle can extract a non-planar orcompound surface directly from a group of points depicting an object inthe 3D point cloud and then implement methods and techniques describedabove to isolate a group of color pixels that depict this surface in theconcurrent 2D color image and to project this group of color pixels ontothe non-planar or compound surface in the 3D point cloud.

2.12 Display: Rider

Block S240 of second method S200 recites rendering points in the first3D point cloud and the first set of synthetic 3D color points on adisplay. Generally, in Block S240, the autonomous vehicle can render theaugmented 3D point cloud (i.e., the current 3D point cloud and augmentedwith a set of synthetic 3D color points) on a display arranged insidethe autonomous vehicle in near real-time, as described above in thefirst method S100.

As described above, the autonomous vehicle can selectively augmentdynamic (and static) objects of interest in the 3D point cloud withsynthetic 3D color points such that these objects are rendered at apoint density greater than other lower-priority objects. For example,the autonomous vehicle can thus present its surrounding field to a riderwith objects depicted at a resolution proportional to importance,thereby enabling the rider to build confidence in the autonomousvehicle's perception of its environment while limiting computationalload necessary to generate and render this representation of the field.In this example, the autonomous vehicle can thus augment nearbyvehicles, pedestrians, and road signs depicted in the 3D point cloudwith synthetic 3D color points derived from the concurrent 2D colorimage but depict other objects—such as trees buildings, the roadsurface, at other vehicles at long distances from the autonomous vehicleat the native resolution of the 3D depth sensor.

The autonomous vehicle can repeat this process to selectively augmenteach subsequent 3D point cloud recorded by the autonomous vehicle.

2.12.1 Image Enhancement

In one variation, the autonomous vehicle enhances an object depicted inthe augmented 3D point cloud rendered on the interior display—withdigital content. In one example, the autonomous vehicle: detects acluster of points in the current 3D point cloud that represent aroadside billboard in Block S220; isolates a cluster of pixels in theconcurrent 2D color image that depict this billboard in Block S230;augments the current 3D point cloud with a set of synthetic 3D colorpoints depicting the roadside billboard in Block S232; and writes colorvalues from nearby synthetic 3D color points to original depth mappoints in this cluster such that the 3D point cloud depicts thebillboard in color and in greater resolution (i.e., greater pointdensity). In this example, the autonomous vehicle can then implementcomputer vision techniques to extract iconography (e.g., text, icons,characters, image features) on the roadside billboard from the 2D colorimage (i.e., from a cluster of color pixels) or from the augmented 3Dpoint cloud (i.e., the cluster of original depth map points and the setof synthetic 3D color points). The autonomous vehicle can then: retrieveelectronic content related to the roadside billboard based on theseiconography; render the augmented 3D point cloud on the interiordisplay; and overlay the electronic content related to the roadsidebillboard over the first 3D point cloud and the first set of synthetic3D color points rendered on the display. In this example, the autonomousvehicle can: retrieve a hyperlink for an advertiser or productadvertised on the billboard from a remote database or computer network;highlight the billboard in the augmented 3D point cloud rendered on theinterior display, such as with a virtual colored border or an animatedtag; and then open a browser window navigated to this hyperlink on theinterior display responsive to the rider selecting the billboard oranimated tag. Similarly, if the autonomous vehicle determines that thebillboard is an advertisement for a local business (e.g., a coffee shop,a retail store) based on iconography detected on the billboard, theautonomous vehicle can render a prompt—to reroute the autonomous vehicleto this local business—over the billboard depicted in the augmented 3Don the interior display; accordingly, the autonomous vehicle can updateits navigation path and reroute to a known location of this localbusiness responsive to the user selecting this prompt on the interiordisplay. Alternatively, the autonomous vehicle can: retrieve a graphicalaugmented reality animation for visual content detected on the billboardfrom a remote database or computer network; and then render thisgraphical augmented reality animation over the billboard depicted in theaugmented 3D on the interior display.

The autonomous vehicle can implement similar methods and techniques toenhance iconography detected on building facades, other vehicles (e.g.,advertisements or logos on semi-trailers), and road signs depicted in 3Dpoint clouds rendered on the interior display. However, the autonomousvehicle can selectively enhance a 3D point cloud with external contentin any other way before rendering this enhanced 3D point cloud on theinterior display inside the autonomous vehicle.

2.12.2 Display: Remote Operator

Additionally or alternatively, the autonomous vehicle can serve theaugmented 3D point cloud to a remote operator—such as when theautonomous vehicle disengages from autonomous operation—in order toenable the remote operator perceive the scene around the autonomousvehicle in less time and/or with greater accuracy and to assist theautonomous vehicle in electing a next navigational action accordingly.

In this implementation, the autonomous vehicle can execute Blocks ofsecond method S200 to selectively augment objects depicted in thecurrent 3D point cloud responsive to a perception failure of theautonomous vehicle. For example, if the autonomous vehicle disengagedfrom autonomous operation due to failure to perceive a particular objectin the current 3D point cloud, the autonomous vehicle can: execute theforegoing methods and techniques to selectively augment a cluster ofpoints depicting this particular object in the 3D point cloud with colordata from the concurrent 2D color image; and then serve a request forremote assistance and all or a portion of this augmented 3D point cloudto a remote operator portal. The remote operator portal can then renderthis augmented 3D point cloud, including the particular object depictedat enhanced resolution. A remote operator at this remote operate portalmay then review this augmented 3D point cloud to perceive the scenearound the autonomous vehicle —including the particular object thusshown in more detail—before returning a command or navigational actionto the autonomous vehicle.

Therefore, in this implementation, the autonomous vehicle can executeBlocks of the second method S200 to selectively augment a current 3Dpoint cloud with color data before uploading this augmented 3D pointcloud to a remote operator portal of a remote operator selected toremotely assist the autonomous vehicle.

2.12.3 Display: Human Annotator

The autonomous vehicle or a remote computer system can similarlyselectively augment a 3D point cloud with color data from a concurrent2D color image before this augmented 3D point cloud is served to andrendered on a local machine for manual annotation by a human annotator.

2.13 Perception with Augmented 3D Depth Image

In one variation of the second method S200, the autonomous vehicleselectively augments a cluster of points—in the current 3D pointcloud—that represents a surface on an object that the autonomous vehicleidentified with lower confidence in the current or preceding 3D pointcloud. Once the autonomous vehicle has augmented this cluster of pointswith color data from the concurrent 2D color image, the autonomousvehicle can reprocess all of the augmented 3D point cloud or a segmentof the augmented 3D point cloud depicting this object in order toidentify and/or characterize the object with greater confidence. Inparticular, if the autonomous vehicle fails to perceive an object in thefield nearby with sufficient confidence based on data contained in anoriginal 3D point cloud, the autonomous vehicle can selectively augmenta region of the 3D point cloud depicting this object with 2D color imagedata and then reprocess this augmented region of the 3D point cloud inorder to identify and characterize the object with greater confidence inBlock S250. The autonomous vehicle can then elect a next navigationalaction in Block S252 based on characteristics of the object thus derivedfrom the augmented 3D point cloud and autonomously execute this nextnavigational action in Block S254.

In one implementation, after recording a 3D point cloud and a concurrentcolor image during a current scan cycle, the autonomous vehicle passesthe original 3D point cloud through a perception pipeline to identifycharacteristics of objects thus represented by points in the 3D pointcloud. Then, if an initial confidence for characteristics of aparticular object represented in the 3D point cloud is less than athreshold confidence, the autonomous vehicle can: isolate a cluster ofcolor pixels in the concurrent color image that correspond to a clusterof points representing this particular object in the 3D point cloud;augment this cluster of points with synthetic 3D color points generatedbased on this cluster of color pixels; and then pass the first 3D pointcloud—now augmented with the synthetic 3D color points—back through theperception pipeline to identify characteristics of the particular objectwith a revised (e.g., greater) confidence in Block S250. The autonomousvehicle can then elect and execute a next navigational actionaccordingly in Blocks S252 and S254.

2.14 Other Domains

In one variation, the autonomous vehicle can implement similar methodsand techniques to selectively augment a current 3D point cloud with datain other domains and recorded by other sensors on the autonomousvehicle.

For example, the autonomous vehicle can include a RADAR sensor arrangedon the front of and facing outwardly from the front of the autonomousvehicle, configured to detect surfaces in its field of view (i.e., aheadof the autonomous vehicle), and configured to output a list of thesesurfaces and their positions once per scan cycle. For example, the RADARsensor can define an approximately 2D field of view extendinghorizontally and outwardly from the front of the autonomous vehicle.Once per scan cycle (e.g., at a rate of 20 hz), the RADAR sensor canoutput an object list for objects detected in its field of view (e.g.,up to 64 objects), such as including: an azimuthal angle relative to theRADAR sensor, a distance from the RADAR sensor, and a speed relative tothe RADAR sensor (i.e., relative to the autonomous vehicle moregenerally) for each object in the object list. In this example, theautonomous vehicle can: detect a set of dynamic objects in the current3D point cloud; selectively populate groups of points in the current 3Dpoint cloud corresponding to these objects with velocities detected bythe RADAR sensor; render this augmented 3D point cloud on the interiordisplay inside the autonomous vehicle; and render velocity tags linkedto groups of pixels corresponding to these dynamic objects in the 3Dpoint cloud or with these groups of pixels rendered in color valuesbased on speeds of their corresponding dynamic objects.

The systems and methods described herein can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any suitable combination thereof. Othersystems and methods of the embodiment can be embodied and/or implementedat least in part as a machine configured to receive a computer-readablemedium storing computer-readable instructions. The instructions can beexecuted by computer-executable components integrated bycomputer-executable components integrated with apparatuses and networksof the type described above. The computer-readable medium can be storedon any suitable computer readable media such as RAMs, ROMs, flashmemory, EEPROMs, optical devices (CD or DVD), hard drives, floppydrives, or any suitable device. The computer-executable component can bea processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

What is claimed is:
 1. One or more tangible non-transitorycomputer-readable storage media storing computer-executable instructionsfor performing a computer process on a computing system, the computerprocess comprising: accessing a two-dimensional (2D) image capturedusing a camera of a vehicle; accessing a three-dimensional (3D) pointcloud captured using a 3D sensor of the vehicle; generating a 3D virtualenvironment representing a field external to the vehicle by projectingat least a portion of the 2D image onto a group of points in the 3Dpoint cloud, the group of points representing a surface in the fieldexternal to the vehicle; and rendering the 3D virtual environment fordisplay on a remote operator display, the vehicle being remotelyoperated using the 3D virtual environment displayed on the remoteoperator display, wherein the vehicle is remotely operated following atleast one of identification of at least one operational issue of thevehicle or receipt of a request for remote assistance.
 2. The one ormore tangible non-transitory computer-readable storage media of claim 1,the computer process further comprising: determining at least onenavigational action for the vehicle using the 3D virtual environmentdisplayed on the remote operator display; and sending the at least onenavigational action to the vehicle.
 3. The one or more tangiblenon-transitory computer-readable storage media of claim 1, the computerprocess further comprising: identifying a perception failure of thevehicle, the at least one operational issue including the perceptionfailure.
 4. The one or more tangible non-transitory computer-readablestorage media of claim 1, the computer process further comprising:determining an alignment between the camera and the 3D sensor using the3D virtual environment displayed on the remote operator display, atleast one of the camera or the 3D sensor being calibrated based on thealignment.
 5. The one or more tangible non-transitory computer-readablestorage media of claim 1, wherein the 3D virtual environment isgenerated in response to detection of an event encountered by thevehicle, the computer process further comprising: determining whetherthe event affected operation of at least one of the camera or the 3Dsensor using the 3D virtual environment.
 6. The one or more tangiblenon-transitory computer-readable storage media of claim 1, the computerprocess further comprising: remotely diagnosing the at least oneoperational issue of the vehicle using the 3D virtual environmentdisplayed on the remote operator display.
 7. The one or more tangiblenon-transitory computer-readable storage media of claim 1, the computerprocess further comprising: identifying the group of points byapproximating a plane in the 3D point cloud, wherein projecting theportion of the 2D image comprises augmenting the group of points withcolor data from the 2D image.
 8. The one or more tangible non-transitorycomputer-readable storage media of claim 1, the computer process furthercomprising: isolating a group of color pixels representing the surfacein the 2D image, wherein projecting the portion of the 2D imagecomprises projecting the group of color pixels onto the surface in the3D point cloud.
 9. A method comprising: accessing a two-dimensional (2D)image captured using a camera of a vehicle; accessing athree-dimensional (3D) point cloud captured using a 3D sensor of thevehicle; generating a 3D virtual environment representing a fieldexternal to the vehicle by projecting at least a portion of the 2D imageonto a group of points in the 3D point cloud, the group of pointsrepresenting a surface in the field external to the vehicle; displayingthe 3D virtual environment on a remote operator display; and remotelyoperating the vehicle using the 3D virtual environment displayed on theremote operator display, wherein the vehicle is remotely operatedfollowing at least one of identification of an operational issue of thevehicle or receipt of a request for remote assistance.
 10. The method ofclaim 9, wherein the 3D virtual environment includes a first portionrendered with a first resolution and a second portion rendered with asecond resolution, the first resolution being higher than the secondresolution and the first portion having a higher priority than thesecond portion.
 11. The method of claim 9, wherein the 3D virtualenvironment is generated in response to detection of an eventencountered by the vehicle, the method further comprising: remotelydiagnosing the vehicle following the event using the 3D virtualenvironment displayed on the remote operator display.
 12. The method ofclaim 9, wherein remotely operating the vehicle comprises: determiningat least one navigational action for the vehicle using the 3D virtualenvironment displayed on the remote operator display; and sending the atleast one navigational action to the vehicle.
 13. The method of claim 9,further comprising: determining an alignment between the camera and the3D sensor using the 3D virtual environment displayed on the remoteoperator display; and calibrating at least one of the camera or the 3Dsensor based on the alignment.
 14. A system comprising: a remoteoperator system in communication with a vehicle over a network, thevehicle including a camera configured to capture a two-dimensional (2D)image and a three-dimensional (3D) sensor configured to capture a 3Dpoint cloud, the remote operator system having at least one processconfigured to generate a 3D virtual environment representing a fieldexternal to the vehicle by projecting at least a portion of the 2D imageonto a group of points in the 3D point cloud, the group of pointsrepresenting a surface in the field external to the vehicle; and aremote operator display configured to display the 3D virtualenvironment, the vehicle being remotely operated using the 3D virtualenvironment displayed on the remote operator display, wherein thevehicle is remotely operated following at least one of identification ofat least one operational issue of the vehicle or receipt of a requestfor remote assistance.
 15. The system of claim 14, wherein remotelyoperating the vehicle comprises: determine at least one navigationalaction for the vehicle using the 3D virtual environment displayed on theremote operator display; and send the at least one navigational actionto the vehicle.